CLDSRV-968: Expire idle keep-alive sockets in the functional test client by tcarmet · Pull Request #6247 · scality/cloudserver

tcarmet · 2026-08-04T22:37:02Z

The functional test client keeps pooled keep-alive connections indefinitely, so after an idle gap it can send a request over a connection the server has already closed and fail with a spurious connection reset. Because the client also runs with retries disabled, one reset is enough to fail a hook and take a whole group of tests down with it, which surfaces as intermittent CI red on slower full-deployment environments where such idle gaps actually occur.

This makes the client expire its own idle connections before the server does, so the reuse race cannot arise.

bert-e · 2026-08-04T22:37:06Z

Hello tcarmet,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
`/after_pull_request`	Wait for the given pull request id to be merged before continuing with the current one.
`/bypass_author_approval`	Bypass the pull request author's approval	⭐
`/bypass_build_status`	Bypass the build and test status	⭐
`/bypass_commit_size`	Bypass the check on the size of the changeset `TBA`	⭐
`/bypass_incompatible_branch`	Bypass the check on the source branch prefix	⭐
`/bypass_jira_check`	Bypass the Jira issue check	⭐
`/bypass_peer_approval`	Bypass the pull request peers' approval	⭐
`/bypass_leader_approval`	Bypass the pull request leaders' approval	⭐
`/approve`	Instruct Bert-E that the author has approved the pull request.		✍️
`/create_pull_requests`	Allow the creation of integration pull requests.
`/create_integration_branches`	Allow the creation of integration branches.
`/no_octopus`	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
`/unanimity`	Change review acceptance criteria from `one reviewer at least` to `all reviewers`
`/wait`	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
`/help`	Print Bert-E's manual in the pull request.
`/status`	Print Bert-E's current status in the pull request `TBA`
`/clear`	Remove all comments from Bert-E from the history `TBA`
`/retry`	Re-start a fresh build `TBA`
`/build`	Re-start a fresh build `TBA`
`/force_reset`	Delete integration branches & pull requests, and restart merge process from the beginning.
`/reset`	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

bert-e · 2026-08-04T22:37:12Z

Incorrect fix version

The Fix Version/s in issue CLDSRV-968 contains:

None

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

9.3.16
9.4.0

Please check the Fix Version/s of CLDSRV-968, or the target
branch of this pull request.

bert-e · 2026-08-05T15:11:24Z

Request integration branches

Waiting for integration branch creation to be requested by the user.

To request integration branches, please comment on this pull request with the following command:

/create_integration_branches

Alternatively, the /approve and /create_pull_requests commands will automatically
create the integration branches.

tcarmet · 2026-08-05T15:45:58Z

/create_integration_branches

bert-e · 2026-08-05T15:46:10Z

Integration data created

I have created the integration data for the additional destination branches.

this pull request will merge bugfix/CLDSRV-968-test-client-idle-socket-timeout into
development/9.3
w/9.4/bugfix/CLDSRV-968-test-client-idle-socket-timeout will be merged into development/9.4

The following branches will NOT be impacted:

development/7.10
development/7.4
development/7.70
development/8.8
development/9.0
development/9.1
development/9.2

You can set option create_pull_requests if you need me to create
integration pull requests in addition to integration branches, with:

@bert-e create_pull_requests

The following options are set: create_integration_branches

bert-e · 2026-08-05T15:46:11Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

The following options are set: create_integration_branches

bert-e · 2026-08-06T17:44:15Z

Incorrect fix version

The Fix Version/s in issue CLDSRV-968 contains:

9.3.16
9.4.0

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

9.3.17
9.4.0

Please check the Fix Version/s of CLDSRV-968, or the target
branch of this pull request.

The following options are set: create_integration_branches

The SDK agents in the aws-node-sdk test client enable keepAlive without an idle timeout, so pooled sockets are never expired client-side. Node's HTTP server closes idle keep-alive connections after 5s and advertises `Keep-Alive: timeout=5`, but the agent only honours that hint when it already has a non-zero timeout of its own. Without one, a burst of requests issued after an idle period can be written to connections the server has already closed, failing with ECONNRESET ("socket hang up"). Set a 4s agent idle timeout so the client retires sockets before the server does. This only destroys sockets sitting in the free pool; in-flight requests stay governed by requestTimeout.

tcarmet · 2026-08-06T17:46:33Z

/reset

bert-e · 2026-08-06T17:46:42Z

Reset complete

I have successfully deleted this pull request's integration branches.

The following options are set: create_integration_branches

bert-e · 2026-08-06T17:47:42Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

The following options are set: create_integration_branches

tcarmet · 2026-08-06T17:48:22Z

/approve

bert-e · 2026-08-06T18:13:18Z

Build failed

The build for commit did not succeed in branch w/9.4/bugfix/CLDSRV-968-test-client-idle-socket-timeout

The following options are set: approve, create_integration_branches

bert-e · 2026-08-06T18:21:49Z

Build failed

The build for commit did not succeed in branch bugfix/CLDSRV-968-test-client-idle-socket-timeout

The following options are set: approve, create_integration_branches

codecov · 2026-08-06T20:00:28Z

❌ 1 Tests Failed:

Tests completed	Failed	Passed	Skipped
8514	1	8513	0

View the top 1 failed test(s) by shortest run time

should handle abort during concurrent CompleteMPU without corruption::Abort MPU - Race Conditions With v4 signature should handle abort during concurrent CompleteMPU without corruption

Stack Traces | 0.109s run time

No MPU metadata should remain

1 !== 0

View the full list of 1 ❄️ flaky test(s)

"after each" hook for "should handle abort during concurrent CompleteMPU without corruption"::Abort MPU - Race Conditions With v4 signature "after each" hook for "should handle abort during concurrent CompleteMPU without corruption"

Flake rate in main: 100.00% (Passed 0 times, Failed 7 times)

Stack Traces | 0.032s run time

The specified multipart upload does not exist. The upload ID might be invalid, or the multipart upload might have been aborted or completed.

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

bert-e · 2026-08-06T20:00:40Z

Build failed

The build for commit did not succeed in branch w/9.4/bugfix/CLDSRV-968-test-client-idle-socket-timeout

The following options are set: approve, create_integration_branches

bert-e · 2026-08-10T20:15:57Z

Build failed

The build for commit did not succeed in branch bugfix/CLDSRV-968-test-client-idle-socket-timeout

The following options are set: approve, create_integration_branches

bert-e · 2026-08-10T20:54:38Z

In the queue

The changeset has received all authorizations and has been added to the
relevant queue(s). The queue(s) will be merged in the target development
branch(es) as soon as builds have passed.

The changeset will be merged in:

✔️ development/9.3
✔️ development/9.4

The following branches will NOT be impacted:

development/7.10
development/7.4
development/7.70
development/8.8
development/9.0
development/9.1
development/9.2

This pull request does not target the following hotfix branch(es) so they
will be left untouched:

hotfix/7.10.27
hotfix/7.4.1
hotfix/9.2.36
hotfix/7.10.3
hotfix/7.70.45
hotfix/7.10.30
hotfix/7.6.0
hotfix/7.4.3
hotfix/7.70.51
hotfix/7.4.6
hotfix/6.4.7
hotfix/7.4.9
hotfix/7.10.49
hotfix/7.2.0
hotfix/7.10.2
hotfix/7.4.5
hotfix/9.0.32
hotfix/7.70.11
hotfix/8.8.45
hotfix/7.10.28
hotfix/7.4.0
hotfix/7.4.8
hotfix/7.10.4
hotfix/7.4.2
hotfix/7.4.10
hotfix/7.8.0
hotfix/7.10.8
hotfix/7.9.0
hotfix/7.70.73
hotfix/9.0.7
hotfix/7.4.7
hotfix/7.10.15
hotfix/7.10.0
hotfix/7.4.4
hotfix/9.2.24
hotfix/7.70.21
hotfix/7.10.1
hotfix/7.7.0

There is no action required on your side. You will be notified here once
the changeset has been merged. In the unlikely event that the changeset
fails permanently on the queue, a member of the admin team will
contact you to help resolve the matter.

IMPORTANT

Please do not attempt to modify this pull request.

Any commit you add on the source branch will trigger a new cycle after the
current queue is merged.
Any commit you add on one of the integration branches will be lost.

If you need this pull request to be removed from the queue, please contact a
member of the admin team now.

The following options are set: approve, create_integration_branches

bert-e · 2026-08-10T21:11:41Z

Queue build failed

The corresponding build for the queue failed:

Checkout the status page.
Identify the failing build and review the logs.
If no issue is found, re-run the build.
If an issue is identified, checkout the steps below to remove
the pull request from the queue for further analysis and maybe rebase/merge.

Remove the pull request from the queue

Add a /wait comment on this pull request.
Click on login on the status page.
Go into the manage page.
Find the option called Rebuild the queue and click on it.
Bert-E will loop again on all pull requests to put the valid ones
in the queue again, while skipping the one with the /wait comment.
Wait for the new queue to merge, then merge/rebase your pull request
with the latest changes to then work on a proper fix.
Once the issue is fixed, delete the /wait comment and
follow the usual process to merge the pull request.

bert-e · 2026-08-10T22:17:45Z

I have successfully merged the changeset of this pull request
into targetted development branches:

✔️ development/9.3
✔️ development/9.4

The following branches have NOT changed:

development/7.10
development/7.4
development/7.70
development/8.8
development/9.0
development/9.1
development/9.2

Please check the status of the associated issue CLDSRV-968.

Goodbye tcarmet.

tcarmet marked this pull request as ready for review August 4, 2026 23:08

tcarmet requested a review from a team August 4, 2026 23:08

leif-scality approved these changes Aug 5, 2026

View reviewed changes

tcarmet requested a review from a team August 5, 2026 15:45

anurag4DSB approved these changes Aug 5, 2026

View reviewed changes

jonathan-gramain reviewed Aug 5, 2026

View reviewed changes

Comment thread tests/functional/aws-node-sdk/test/support/config.js

BourgoisMickael approved these changes Aug 5, 2026

View reviewed changes

Comment thread tests/functional/aws-node-sdk/test/support/config.js Outdated

tcarmet force-pushed the bugfix/CLDSRV-968-test-client-idle-socket-timeout branch from 0065a20 to 7346130 Compare August 6, 2026 17:46

scality deleted a comment from bert-e Aug 6, 2026

scality deleted a comment from bert-e Aug 10, 2026

bert-e merged commit 24fb6b3 into development/9.3 Aug 10, 2026
87 of 107 checks passed

bert-e deleted the bugfix/CLDSRV-968-test-client-idle-socket-timeout branch August 10, 2026 22:17

Uh oh!

Conversation

tcarmet commented Aug 4, 2026

Uh oh!

bert-e commented Aug 4, 2026

Hello tcarmet,

Uh oh!

bert-e commented Aug 4, 2026

Incorrect fix version

Uh oh!

bert-e commented Aug 5, 2026

Request integration branches

Uh oh!

tcarmet commented Aug 5, 2026

Uh oh!

bert-e commented Aug 5, 2026

Integration data created

Uh oh!

bert-e commented Aug 5, 2026

Waiting for approval

Uh oh!

Uh oh!

Uh oh!

bert-e commented Aug 6, 2026

Incorrect fix version

Uh oh!

tcarmet commented Aug 6, 2026

Uh oh!

bert-e commented Aug 6, 2026

Reset complete

Uh oh!

bert-e commented Aug 6, 2026

Waiting for approval

Uh oh!

tcarmet commented Aug 6, 2026

Uh oh!

bert-e commented Aug 6, 2026

Build failed

Uh oh!

bert-e commented Aug 6, 2026

Build failed

Uh oh!

codecov Bot commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ 1 Tests Failed:

Uh oh!

bert-e commented Aug 6, 2026

Build failed

Uh oh!

bert-e commented Aug 10, 2026

Build failed

Uh oh!

bert-e commented Aug 10, 2026

In the queue

Uh oh!

bert-e commented Aug 10, 2026

Queue build failed

Uh oh!

bert-e commented Aug 10, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

codecov Bot commented Aug 6, 2026 •

edited

Loading